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REMARKS 

This paper is submitted in response to the Office Action 
for the above- identified application mailed August 27, 2003. 

In the Office Action, all the claims , Claims 1-16/ were 
rejected under 35 U.S.C. Sec. 102 for being anticipated by 
U.S. Patent No. 6 259 827 to Nichani . 

Claim 9 was rejected under 35 U.S.C. Sec 112 for failing 
to particularly point out and distinctly claim the invention 
to which it is directed. 

An objection was lodged against the title of this 
application. 

Before discussing this Response, the Applicants direct 
the Examiner's attention to a minor administrative matter for 
which the assistance of the Examiner is courteously requested. 
On May 21, 2001 the Applicants filed an Information Disclosure 
Statement for this application with the Patent and Trademark 
Office. A single document was cited under this paper. The 
Examiner acknowledged considering this document. However, on 
the Form PTO-1449 under which this document was submitted, the 
document was inadvertently identified as a United Kingdom 
patent publication. The document is, in fact a European 
patent publication. Enclosed herewith is a substitute Form 
PTO-1449 in which the national origin of the document is 

• properly identified. The Applicants request the Examiner (1) 
indicate on the previously submitted Form PTO-144 9 the 

# inaccurately identified document was not considered; and (2) 

initial the current Form PTO-144 9 to acknowledge consideration 
of the previously considered document. 

The Examiner has already acknowledged consideration of 
the cited document. Therefore, it is submitted that the 
reacknolwedgement of the consideration of the document does 
not impose any burden on the Office or the Examiner. The 
Applicants and their representatives thank the Examiner for 
this courtesy . 
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In this Response, the Applicants have adopted the title 
suggested by the Examiner. 

Also under cover of this Response, the Applicants submit 
marked-up and substitute versions of the specification. The 
changes the Applicants propose to make to the specification as 
filed are presented on the marked-up specification. These 
changes include the insertion of the section headers normally 
present in United States patent applications and certain other 
editorial changes . The proposed changes are entered on the 
substitute specification. The undersigned registered 
representative hereby states that the substitute specification 
does not contain any new matter. 

Also under cover of this Response, Claims 1 and 9 are 
amended, Claims 2-8 and 10-16 are cancelled and new 
Claims 17-34 are added. Claims 1 and 9 remain the independent 
claims of this application. Claims 17-24 are dependent from 
Claim 1. Claims 25-34 are dependent from Claim 9. 

Claim 9, as does each claim present in this application, 
particularly points out and distinctly claims the invention to 
which it is directed. Therefore, it is requested that the 
rejection directed to the failure to satisfy this statutory 
standard be withdrawn. 

Nichani is directed to a method in which images of an 
object and of the object and its background are generated, and 
the second image is subtracted from the first image to form a 
third image. 1 

The above method is not the same as the Applicants' 
method and apparatus of Claims 1 and 9 wherein, from a digital 
representation of the image, predetermined objects are 
identified and located and data representing the boundaries 
between regions on the image having different characteristics 
are identified. Then, based on the locations of the objects 
and the boundary data, the objects are grouped into clusters. 



1 U.S. Patent No. 6,259,827, column 3, lines 28-42. 
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Nichani's system is designed to "increase [] a contrast 
between the object and the background." 2 This is not the same 
end result one gets by practicing the Applicants' claimed 
invention, namely the identification of clusters of objects 
that have been grouped as a function of both the proximity of 
the objects to each other and the boundaries between the image 
regions that have different characteristics. 

Thus, the prior art simply fails to suggest an assembly 
or method having either the features or benefits of 
Applicants' invention as recited by Claims 1 and 9. 
Therefore, it is submitted that these claims are directed to a 
patentable invention and are in an allowable form. 

The dependent claims are all allowable at least because 
they depend from allowable independent claims. 

Moreover, Claims 23, 24, 31 and 32 are directed to the 
version of Applicants' invention wherein the clusters are 
formed so that only objects that do not cross a region 
boundary are potentially grouped together. This essentially 
eliminates the likelihood that objects that are in different 
regions, and therefore should not be grouped together, are 
inadvertently grouped together. 

Given that the prior art fails to disclose anything about 
grouping objects, it can hardly be said to suggest the above - 
claimed invention that prevents the cross-region grouping of 
objects . 

Therefore, at least the above-discussed dependent claims 
are further in condition for allowance because they are 
independently directed to an invention entitled to patent 
protection. 

In conclusion, it is respectfully submitted that all the 
claims of this application are directed to a patentable 
invention and are in an allowable form. Since the claims, as 
well as the other parts of this application, are in an 



2 U.S. Patent No. 6,259,827, column 3, lines 43-48. 
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allowable state, the Applicants now courteously solicit prompt 
issuance of a Notice of Allowance. 

Respectfully submitted, 



David S . Goldenberg 

DSG\pcq 
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IMAGE PROCESSING 



Field of the Invention 



This invention relates to image processing and is of 
particular benefit to clustering data in binary images. 



Background of the Invention 



In a number of image processing applications there is a 
requirement to cluster objects together, i.e. objects which 
are related due to their close proximity to each other. For 
example in document image processing, text objects that are 
close to each other may be clustered together to form 
paragraphs. With binary image data this clustering can be 
performed by merging, i.e. removing gaps between objects 
that are less than a specified distance limit. The 
difficulty in merging binary objects into clusters is the 
determination of the distance limit. If the distance limit 
is too small then some related objects remain separate. If 
the distance limit is too large then gaps between clusters 
may be removed incorrectly. 

If the separation between clusters is significantly larger 
than the gaps between objects within a cluster, then the 
setting of the distance limit is easy. In the cases where 
the separation between clusters is similar to the gaps 
between objects in a cluster, deciding the distance limit 
for merging is more difficult. If the separation between 
clusters is smaller than the gaps between objects in a 
cluster then it may be impossible for a single distance 
limit to merge objects into clusters without also 
incorrectly joining clusters together. In the example of 
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text processing of document images, merging text into 
paragraphs is easy if the separation of the paragraphs is 
significantly larger than the gap between letters. 
However if the gap between letters is less than the 
separation between paragraphs (as is often the case in a 
document with many fonts and sizes of text) , then it may 
not be possible to cluster all the text into paragraphs 
successfully using a simple binary merging operation. 

In image processing applications where the clustering of 
binary objects is difficult because of the close proximity 
of the clusters, it is often helpful to use additional 
information to segment the binary image. The information 
used to segment the binary image is generally more useful 
if taken from a separate source or earlier form of the 
image. In the example of text processing in document 
images, the binary image of the text objects may be 
segmented according to background colour, calculated from 
the original greyscale image of the document . Unfortunately 
segmentation of an image can be difficult and many 
techniques do not adequately account for slowly varying 
features or incomplete region boundaries . 

Summary of the Invention 
We have appreciated that the process of segmenting and 
merging in order to cluster objects in a binary image may 
be made more successful and computationally efficient if 
they are combined into a single process, where the 
segmentation information is represented as the boundaries 
between regions. Accordingly a preferred embodiment of 
the invention clusters together objects in a binary image 
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by removing the gaps between objects in the cases where 
the gaps are less than a specified distance limit and do 
not cross a region boundary. 

We have observed that the merging of objects in a binary 
image into clusters can benefit from segmentation of that 
image. If the segmentation can separate clusters without 
dissecting them then it reduces the likelihood of 
incorrectly merging clusters together. This can simplify 
the requirements of the merging operation making it easier 
to successfully set a distance limit for the merging. We 
have also observed that the requirement of merging objects 
in a binary image into clusters also simplifies the task 
of segmentation. As an isolated task the segmentation 
would need to be able to separate the whole image into 
distinct regions. However the merging operation has a 
distance limit which will keep clusters that are well 
separated isolated. Thus the demands on the segmentation 
are reduced to separating regions where the clusters would 
otherwise be merged together. The benefit of the invention 
performing both the merging and segmentation simultaneously 
is to take advantage of the reduced requirements of the 
segmentation information and the simplification in the 
setting of the merging distance limit. 

The — invention ia defined with more precision in the 
appended claims to which reference 3hould now be made. 

Brief Description of the Drawings 
A preferred embodiment of the invention will now be 
described in detail by way of an example with reference to 
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the accompanying drawings in which: 

Figure 1 shows schematically apparatus for identifying 
addresses on complex envelopes; 

Figure 2 shows schematically a conventional method for 
applying both segmentation and merging to a binary image; 
Figure 3 shows schematically an embodiment of the 
invention used to perform the tasks of both segmentation 
and merging on a binary image; 

Figure — 4 — shows benefit of using regions boundaries — for 
segmentation in the case of segmenting text — in a document 
image according to background colour; — Figures 4A, 4B, 4C 
and 4D are depictions of complete and background images 
scanned according to this invention; 

Figure 4E depicts the border between the background image 
of Figure 4D; and 

Figure 5 shows schematically apparatus for identifying 
addresses on complex envelopes using an embodiment of the 
invention. 

Detailed Description 
In an example of identifying addresses on complex 
envelopes in a mail processing system there is a 
requirement to cluster text objects into paragraphs that 
may be addresses. As with many other image processing 
applications the merging of binary text objects into 
paragraphs can benefit from the use of additional 
information to segment the image. Accordingly in the case 
of processing document images such as mail we have 
proposed in our British Patent Application No. -£ 
j-GB 2 3 64 416 A (Corresponding PCT Pub. No. 
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WO 02/03315 Al) filed on the same day as the current 
application, a method of clustering related text objects in 
an image document. The aforementioned patent uses two 
segmentations, one according to text colour and one 
according to background colour, to segment the binary image 
of the text objects. In the example shown Figure 1 the 
apparatus to identify addresses on complex envelopes, uses 
a simplified version that creates a segmentation from the 
background information only. 

In Figure 1, a scanner 2 produces a 256 level greyscale 
image from the input document. It could of course produce 
other resolutions of greyscale image or indeed a colour 
image. This greyscale image then passes to a text object 
extraction unit 4 which identifies the text objects in the 
image and produces a binary image of thcgc the text objects . 

At the same time, the greyscale image passes to a global 
information segmentation unit 6 which creates a 
segmentation for the image based on the background 
greyscale level by defining regions where the grey level is 
the same or within a predetermined range. This could 
also use colour information. This segmentation data 
relating to background grey level is then passed to a 
segmentation unit 8 which also receives the binary image 
from the text object extraction unit 4. g^yre Segmentation 
Unit 8 then segments the binary image of text according to 
the global background information supplied to it. 

The output data for thio from segmentation unit 8 is then 
passed to the merging unit 10 which, for each segmented 
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region, merges text objects according to the distance 
between letters . Letters which are separated by less than a 
predetermined distance are merged together and ones which 
are further apart are not. This produces text blocks which 
are passed to a sorting unit 12. This sorts the text blocks 
according to those most likely to contain an address (in 
the case of a mail processing system) and passes these to 
an optical character recognition (OCR) unit 14. 

Steps 8 and 10 in Figure 1 are a binary segmentation and 
merger. A conventional approach to both segmentation and 
merging of a binary image is shown in Figure 2 . This 
requires the segmentation to separate the image into 
complete regions. The merging operation can then be applied 
to each region separately. In Figure 2 a segmentation map 
is created from segmentation information by a segmentation 
mapping unit 16. This map of segmented regions is combined 
with the binary image (of text or other objects) in a 
binary image segmentation unit 18. Thio Binary image 
segmentation unit 18 outputs, in turn, separated regions of 
the binary image to a merging unit 2 0 wh i c h me r g i ng un it 2 0 
removes gaps between objects below a specified distance 
limit with each region. The output of -fefejr amerging unit 2 0 
is then a sequence of clustered binary objects which do not 
cross region boundaries . The clusters are constrained by 
the distance limit of the merging applied. The task of 
using the segmentation information to separate the image 
into distinct regions is difficult to do accurately and 
therefore this system does have drawbacks. 

An embodiment of the invention as shown in Figure 3 is an 
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alternative to this. Instead of attempting to use the 
segmentation information to separate the image into 
complete regions, the task is reduced to extracting the 
region boundaries. Removing gaps between objects in the 
binary image then performs the merging operation. However, 
gaps are only removed if they are less than the specified 
distance limit and do not cross a region boundary. The 
result is effectively a simultaneous segmentation and 
merging operation with the advantage that at no point does 
the image have to be completely segmented. In fact, since 
the region boundaries do not have to be complete, the 
process is significantly simpler than forming an isolated 
segmentation . 

The system of Figure 3 comprises a boundary extraction unit 
22 which extracts region boundaries from segmentation 
information derived from the input greyscale image or some 
other source. This segmentation information may be 
information defining changes in background colour or 
greyscale which exceeds a predetermined limit. This gives a 
binary image of region boundaries which forms one of the 
inputs to the merging and segmentation unit 24 . The other 
input to this is the binary image, which in the case of 
mail processing would come from the text object extraction 
25 unit 4 of Figure 1. 

The segmentation and merging unit 24 then merges together 
objects in the binary image without crossing any region 
boundaries. Merging is performed by removing gaps between 
objects that are below the merging distance limit and 
which do not cross a region boundary. Normally, such a 
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process will be performed by conventional hardware 
programmed with software to perform this process. 
However, dedicated hardware circuitry could be provided to 
implement the segmentation and merging. The benefit of 
using boundaries for segmentation is illustrated in Figure 
4-a-) — e4-= — Figure 4 a) — and b) Figures 4A-4E. figures 4A and 4B 
show the original and background to a document in a mail 
processing system. In order to aid the clustering of text 
objects in the paragraphs. The background has been 
extracted from the original image to segment the binary 
text obj cts obj ects . This segmentation provided by the 
background is shown in Figure 4fe4B. As can be seen, the 
background areas are obviously distinct and there is no 
difficulty in separating the image into complete distinct 
regions according to background. 

Figures 4-e-)-4C and dr)-4D show original and background of 
another envelope. The top half of the image is a table of 
figures and the bottom half has some text. However, the 
text on the lower section is printed on a background which 
darkens from right to left. This gradient makes it 
difficult to segment the image into distinct regions. On 
the left side, it is clear that the image has upper and 
lower parts which should be separated. On the right there 
is no apparent boundary. Complete segmentation would 
therefore be very difficult. However, region boundaries are 
simple to extract from the background and give a good 
accurate representation of where merging can and cannot 
occur. Thus, the region boundaries can be extracted from 
the background information and these are illustrated in 
Figure 4e^-4E. In this, the line dividing upper and lower 
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parts is distinct in the left-hand portion of the image but 
gradually decreases in distinctness towards the right-hand 
edge. Using the system of Figure 3, no merging can take 
place across this boundary. 

A further advantage of this simultaneous merging and 
segmentation is that, whereas normal segmentation 
information needs to be able to segment the whole image, in 
this particular system, it only needs to represent a 
region boundary. This can be merely a line. It does not 
need to enclose the distinct region. When used with 
document image processing, the text objects, the 
background colour, text colour, text orientation, etc., 
can all be used to segment the whole image. However, with 
the current technique, incomplete boundaries such as bold 
lines, location of images and logos, etc., can all be used 
successfully to aid clustering of text objects. In 
addition, repetitive segmentations are normally 
computationally intensive. The present technique requires 
only a binary image of the lines not to be crossed during 
merging. Thus, the multiple segmentations represent a 
case of Or-ing a number of binary images to create a 
complete binary image of the region boundaries. This is 
computationally much less intensive. 

The embodiment shown in Figure 5 shows the merging and 
segmentation system of Figure 3 used in the apparatus of 
Figure 1 and replacing the segmentation unit 8 and the 
merging unit 10 . The segmentation information input to 
the boundary extraction units 22 comes from the global 
segmentation unit 6. The binary image input to the merging 
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and segmentation unit comes from the text object extraction 
unit 4. The output of the segmentation and merging unit 24 
is a set of text objects grouped into text blocks. These 
are then sorted for the text block most likely to be the 
address by the sorting unit 12 before passing to the OCR 
14. The system can easily be implemented to use text colour 
segmentation instead of greyscale segmentation. 

The invention performs the clustering of objects in a 
binary image where the clusters are described by a maximum 
distance allowed between objects and by some information 
that implies segmentation between clusters. The 
segmentation information is supplied as a binary image of 
region boundary lines that are not crossed during the 
clustering operation. The region boundaries do not have 
to be complete. Since the invention is a general image 
processing technique for clustering of binary objects 
there are numerous applications. The main example used in 
this description has been the clustering of text objects 
into paragraphs in document images. Other applications 
could include : 

• Biological Image Processing 

E.g. Clustering binary image of foreign bodies 
infecting cells, where the segmentation region 
boundaries are the cell walls. 

• Geographical Image Processing 

E.g. Clustering live stock in images of fields where 
the region boundaries are the hedge rows . 
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• Image Processing in military applications 

E.g. Clustering of personnel in a building, where the 
binary image of personnel locations may be taken from 
a thermal imaging system and the segmentation region 
boundaries are taken as the walls in the building 
plans . 

What is claimed is: 



